Picture encoding method and picture decoding method

ABSTRACT

A picture encoding method of the present invention is a picture encoding method of predictively encoding an input picture with reference to pictures stored in a picture buffer, decoding the encoded input picture, judging whether or not the decoded picture is a picture for reference and whether or not the decoded picture is a picture for output which needs to be stored until its display time, and storing, in the picture buffer, the picture for reference and the picture for output based on the determination result.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a divisional of application Ser. No. 13/461,237,filed May 1, 2012, which is a continuation of application Ser. No.11/976,272, filed Oct. 23, 2007, now U.S. Pat. No. 8,189,680, which is acontinuation of application Ser. No. 11/605,958, filed Nov. 30, 2006,now U.S. Pat. No. 7,995,658, which is a divisional of application Ser.No. 10/481,012, now U.S. Pat. No. 7,206,347, which is the National Stageof International Application No. PCT/JP03/08576, filed Jul. 7, 2003. Theentire disclosures of the above-identified applications, including thespecification, drawings and claims are incorporated herein by referencein their entirety.

TECHNICAL FIELD

The present invention relates to a picture encoding method forcompressing efficiently a moving picture and a picture decoding methodfor decoding correctly the encoded picture and displaying it.

BACKGROUND ART

Recently, with an arrival of the age of multimedia which handlesintegrally audio, video and pixel values of others, existing informationmedia, i.e., newspapers, journals, TVs, radios and telephones and othermeans through which information is conveyed to people, has come underthe scope of multimedia. Generally speaking, multimedia refers tosomething that is represented by associating not only with charactersbut also with graphics, audio and especially pictures and the liketogether. However, in order to include the aforementioned existinginformation media in the scope of multimedia, it appears as aprerequisite to represent such information in digital form.

However, when calculating the amount of information contained in each ofthe aforementioned information media as the amount of digitalinformation, the information amount per character requires 1˜2 byteswhereas the audio requires more than 64 Kbits (telephone quality) persecond and when it comes to the moving picture, it requires more than100 Mbits (present television reception quality) per second. Therefore,it is not realistic to handle the vast information directly in thedigital format via the information media mentioned above. For example, avideophone has already been put into practical use via IntegratedServices Digital Network (ISDN) with a transmission rate of 64Kbit/s˜1.5 Mbit/s. However, it is not practical to transmit videocaptured on the TV screen or shot by a TV camera. This thereforerequires information compression techniques, and for instance, in thecase of the videophone, video compression techniques compliant withH.261 and H.263 standards internationally standardized by ITU-T(International Telecommunication Union-Telecommunication StandardizationSector) to be employed. According to information compression techniquescompliant with the MPEG-1 standard, picture information as well as musicinformation can be stored in an ordinary music CD (Compact Disc).

Here, MPEG (Moving Picture Experts Group) is an international standardfor compression of moving picture signals and MPEG-1 is a standard thatcompresses video signals down to 1.5 Mbit/s, that is, to compressinformation of TV signals approximately down to a hundredth of theiroriginal size. The transmission rate within the scope of the MPEG-1standard is limited primarily to about 1.5 Mbit/s, therefore, MPEG-2which was standardized with the view to meet the requirements ofhigh-quality picture allows data transmission of moving picture signalsat a rate of 2˜15 Mbit/s. In the present circumstances, a working group(ISO/IEC JTC1/SC29/WG11) in charge of the standardization of the MPEG-1and the MPEG-2 has achieved a compression rate which goes beyond whatthe MPEG-1 and the MPEG-2 have achieved, realized encoding/decodingoperations on a per-object basis and standardized MPEG-4 in order torealize a new function required by the era of multimedia. In the processof the standardization of the MPEG-4, the standardization of an encodingmethod for a low bit rate was aimed. However, the aim is presentlyextended to a more versatile encoding of moving pictures at a high bitrate including interlaced pictures.

Recently, a new picture encoding as a next generation encoding of theMPEG-4 called JVC is under the process of standardization and beingjointly worked on by the ITU-T and the ISO/IEC.

FIG. 24 is a diagram showing a prediction structure, a decoding orderand a display order of pictures. “Picture” is a term indicating either aframe or a field and the term “picture” here is used instead of frame orfield in the present specification. The hatched pictures in FIG. 24represent the pictures to be stored in the memory for reference whenother pictures are encoded/decoded.

I0 is an intra coded picture and P3, P6 and P9 are predictive codedpictures (P-picture). The predictive encoding in the scheme of the JVTstandard differs from that of the conventional MPEG-1/2/4. An arbitrarypicture is selected out of a plurality of encoded pictures as areference picture and a predictive image can be generated from thereference picture. For example, a picture P9 may select an arbitrarypicture out of three pictures of I0, P3 and P6 and generate a predictiveimage using the selected picture. Consequently, it heightens apossibility to select the more applicable predictive image than theconventional case of applying MPEG-1/2/4 and thereby improves acompression rate. B1, B2, B4, B5, B7 and B8 are bi-directionallypredictive coded pictures (B-picture), differing from inter-pictureprediction, wherein a plurality of pictures (two pictures) are selectedand a predictive image is generated using the selected pictures and thenencoded. It is especially known that the accuracy of the predictiveimage can be greatly improved and so can be the compression rate byperforming interpolation prediction using an average value of twopictures temporally previous and subsequent for generating a predictiveimage. The marks of “I” for an intra coded picture, “P” for a predictivecoded picture and “B” for a bi-directionally predictive coded pictureare used in order to differentiate encoding method of each picture.

In order to refer to the temporally previous and subsequent pictures forthe B-pictures, the temporally previous pictures shall be coded/decodedat first. This is called reordering of pictures and often takes place inthe conventional MPEG-1/2/4. Therefore, in contrast with an encodingorder (Stream Order), an order of displaying the pictures which aredecoded (Display Order) is reordered as shown in FIG. 24 showing aprediction structure, a decoding order and a display order of pictures.B-pictures in the example of FIG. 24 are displayed at the moment whenthe stream is decoded, therefore, there is no need to store them whenthey are not referred to by other pictures. However, I-pictures andP-pictures have to be stored in a memory since they are displayed afterbeing decoded when the decoding of the following B-picture isterminated.

The terms and the meanings of the hatched pictures in the diagramshowing the prediction structure, the decoding order and the displayorder of the pictures are the same as those used in FIG. 24.

FIG. 26 is a block diagram showing a picture encoding apparatus forrealizing a conventional picture encoding method. The followingillustrates an operation of the picture encoding apparatus for realizingthe conventional picture encoding method in FIG. 26.

A picture structure determination unit PicStruct determines an encodingtype (I-picture, P-picture and B-picture) for each picture, notifies areference picture control unit RefPicCtrl of the encoding type and thepictures that can be referred to in the encoding and informs also areordering unit ReOrder of the encoding order of the pictures. Thereordering unit ReOrder reorders the order of an input picture PicIninto an encoding order and outputs the reordered pictures to a motionestimation unit ME and a subtraction unit Sub. The motion estimationunit ME refers to the reference pictures stored in a picture memoryPicMem1, determines an applicable reference picture and detects a motionvector indicating a pixel position of the reference picture and sendsthem to a variable length coding unit VLC, the picture memory PicMem1and a motion compensation unit MC. The picture memory PicMem1 outputsthe pixels of the reference picture according to the motion vector MV tothe motion compensation unit MC whereas the motion compensation unit MCgenerates a predictive image using the pixels in the reference picturegained from the picture memory PicMem1 and the motion vector MV.

The subtraction unit Sub calculates a difference between the picturereordered by the reordering unit ReOrder and the predictive image. Thedifference is converted to frequency coefficients by an orthogonaltransformation unit T and then the frequency coefficients are quantizedby the quantization unit Q and outputted as quantized values Coef.

An inverse quantization unit IQ inverse quantizes the quantized valuesCoef and restores them as frequency coefficients. The inverse orthogonaltransformation unit IT performs inverse frequency conversion for thefrequency coefficients to be outputted as pixel differential values. Anaddition unit Add adds the predictive image to the pixel differentialvalues and obtains a decoded picture.

The reference picture control unit RefPicCtrl, according to the encodingtype of the picture, judges whether or not the decoded picture is to bestored in the picture memory PicMem1 to be referred to as a referencepicture and whether or not the decoded picture is to be removed from thepicture memory PicMem1 (no longer referred to as a reference picture)and notifies of the operation using a memory control command MMCO.

A switch SW is turned ON when the memory control command MMCO orders astorage and thereby the decoded picture is stored in the picture memoryPicMem1 as a reference picture. The picture memory PicMem1 releases thearea where the decoded picture is stored so that other decoded picturescan be stored when the picture memory PicMem1 instructs that the decodedpicture shall be removed from the picture memory PicMem1.

The variable length coding unit VLC encodes the quantized values Coef,the motion vector MV and the memory control command MMCO and outputs anencoded stream Str.

The case in which the encoding includes the frequency conversion and thequantization is shown. However, the encoding may be one without them,such as DPCM, ADPCM, and linear predictive encoding. The encoding may beone in which the frequency conversion and the quantization areintegrated or one that is not accompanied by the quantization after thefrequency conversion, as in bit-plane encoding.

FIG. 27 shows bit streams of the memory control command MMCO. Thevariable length coding unit VLC encodes “000” which means a release of awhole memory area so that the picture memory is initialized at thebeginning of the encoding/decoding or in the head of the GOP (Group OfPicture). Also, the variable length coding unit VLC encodes “01” whenthe decoded picture is stored in the picture memory. When a picturestored in the picture memory is released at the same time, the variablelength coding unit VLC encodes a picture number following the “001”since the picture number to be released has to be indicated. When aplurality of pictures are released, the command to release a pictureneeds to be encoded for a plural number of times, therefore, a commandto store a picture is encoded in addition to the command to release apicture. The variable length coding unit VLC encodes sequentially aplurality of memory control commands MMCO and encodes lastly “1”indicating that the memory control command MMCO is complete. In thisway, the memory control command MMCO is encoded as an encoded streamStr.

FIG. 28 is a block diagram showing a picture decoding apparatus forrealizing a conventional picture decoding method. The same numbers areused for the devices that operate in the same manner as the pictureencoding apparatus for realizing the conventional picture encodingmethod shown in FIG. 26.

A variable length decoding unit VLD decodes an encoded stream Str andoutputs a memory control command MMCO, a motion vector MV and quantizedvalues Coef. A picture time Time is inputted from outside and is asignal for specifying a picture to be displayed. When a picture to bedisplayed is a decoded picture, an output from the adding unit Add isselected at a selector Sel and sent out to a display unit Disp. When apicture to be displayed is a picture stored in the picture memoryPicMem1, it is read out from the picture memory PicMem1, selected at theselector Sel and outputted to a display unit Disp.

As described above, the picture memory PicMem1 outputs, to the motioncompensation unit MC, pixels according to the motion vector MV whereasthe motion compensation unit MC generates a predictive image accordingto the pixels obtained from the picture memory PicMem1 together with themotion vector MV.

The inverse quantization unit IQ inverse quantizes the quantized valuesCoef and restores them as frequency coefficients. Furthermore, theinverse orthogonal transformation IT performs inverse frequencyconversion for the frequency coefficients to be outputted as pixeldifferential values. The addition unit Add adds the predictive image tothe pixel differential values to generate a decoded picture.

The picture memory PicMem1 releases the area in which the decodedpicture is stored so that another decoded picture can be stored.

The example of the decoding including the inverse frequency conversionand the inverse quantization is described above. However, the decodingmay be one without them, such as DPCM, ADPCM and a linear predictiveencoding. The decoding may be one in which the inverse frequencyconversion and the inverse quantization are integrated or one that isnot accompanied by the inverse quantization after the frequencyconversion as in a bit-plane encoding.

With the use of the picture decoding apparatus for realizing theconventional picture decoding method shown in FIG. 28, it is obviousthat the combination of the conventional picture encoding types shown inFIGS. 24 and 25 allows for a correct decoding of the encoded stream Strencoded by the picture encoding apparatus for realizing the conventionalpicture encoding method shown in FIG. 26.

The more flexible combination is considered here as a picture encodingtype.

FIG. 1 is a diagram showing a prediction structure, a decoding order anda display order of the pictures, which do not exist in the related art.The prediction structure with respect to B-picture differs in thevicinity of Picture 4 in FIG. 1. Namely, Picture 2 that is a B-pictureis stored in the picture memory to be referred to as a predictive imageof Picture 1 and Picture 3. Consequently, the encoding order and thedisplay order of each picture are as shown in FIG. 1.

Pictures B5 and B6 are B-pictures that are not stored since they are notreferred to in a predictive coding. However, differing from FIG. 24, thedisplay time for the pictures B5 and B6 has not yet come at the timewhen they are decoded since it is the time for other picture to bedisplayed. That is, at the time of decoding the picture B5, the pictureP4 shall be displayed and at the time of decoding the picture B6, thepicture B5 shall be displayed. Since the pictures B5 and B6 are notstored, they cannot be taken out from the picture memory at the displaytime. Therefore, the pictures which are not referred to for predictiveencoding are not stored in the picture memory, therefore, the picturesB5 and B6 cannot be displayed after being decoded with the use of theconventional encoding/decoding method. Namely in the case of not storingthe pictures that are not referred to in predictive encoding as in theexample shown in FIG. 24, only Pictures 1, 2, 4, and 7 can be displayed.

Thus, considering the more flexible combination as a picture encodingtype, it is a problem that the pictures which cannot be displayed afterbeing decoded occurs. It is conceivable to add another picture memoryfor display and store the pictures that are not stored in the picturememory PicMem1 in this picture memory for display so that they can bedisplayed; however, the weak point is that this picture memory requiresa huge memory in this case.

Furthermore, there rises a new problem in the reproduction of a picturein the middle of the stream even if another picture memory for displayis introduced. FIG. 2 is a diagram showing a prediction structure, adecoding order and a display order of pictures. The difference comparingwith FIG. 25 is that the prediction structure in the vicinity of Picture7 becomes completely independent. The pictures following a picture I7are not referred to when the pictures with display time preceding thepicture I7 are encoded/decoded. Therefore, the pictures following thepicture I7 can be encoded correctly if the decoding starts from thepicture I7 and the picture I7 can be reproduced independently. In thisway, the insertion of an I-picture while streaming often takes place.This system to reproduce a picture in the middle of the stream, whichcomplies with MPEG-2, is called GOP (Group Of Picture).

The correspondence of a reproduced picture of the picture decodingapparatus and that of the picture encoding apparatus in the case ofreproducing the picture in the middle of the stream has to be assured,and the easy method is to initialize the whole area of the picturememory. However, Picture 6 is not yet displayed and stored in thepicture memory when Picture 7 is decoded, Picture 6 therefore cannot bedisplayed from the picture memory at its display time if the entirepicture memory is initialized before the display of Picture 6 takesplace.

The object of the present invention therefore is to allow the display ofthe pictures that cannot be displayed after being decoded by taking thememory amount necessary for encoding/decoding of the picture intoconsideration.

SUMMARY OF INVENTION

In order to achieve the above objects, a picture encoding method of thepresent invention of predictively encoding an input picture withreference to pictures stored in a picture buffer and decoding theencoded input picture has been developed. The picture encoding methodcomprises: judging whether or not the decoded picture is for referenceand whether or not the decoded picture is for output, needing to bestored until a display time of the decoded picture; and

storing the decoded picture in the picture buffer as one of thefollowing uses: for reference and for output, based on each result ofthe judgments.

Here, the picture encoding method may comprise the following steps: afirst judgment step of judging whether or not the decoded picture is forreference; a second judgment step of judging whether or not the decodedpicture is for output, needing to be stored until display time of thedecoded picture; and a management step of managing the picture buffer sothat the picture buffer stores the decoded picture judged to be forreference in the first judgment step and the decoded picture judged tobe for output in the second judgment step.

With this structure, the judgment on whether the decoded picture is apicture for output in addition to the judgment on whether the decodedpicture is a picture for reference allows a storage not only of apicture for reference but also of a picture that is not referred to inthe predictive encoding but cannot be displayed or outputted as apicture for output at the time of decoding. Such judgment and buffermanagement take place in the picture encoding to simulate a decodingoperation, therefore, a picture for output can surely be displayed oroutputted at its display time by the decoding apparatus.

Here, the picture buffer may have a specified size, and the pictureencoding method may further comprise: a third judgment step of judgingwhether or not the picture for reference stored in the picture buffer isno longer used as a reference picture; and a fourth judgment step ofjudging whether or not the picture for output stored in the picturebuffer is already outputted. In the management step, an area storing apicture in the picture buffer is reused within a scope of the specifiedsize based on each result of the judgments in the third and fourthjudgment steps.

With this structure, an area storing a picture is reused in the scope ofthe specified size of the picture buffer for storing the picture forreference and the picture for output in the management step, therefore,the picture buffer may be adapted for the specified size of essentialrequirements without wasting the memory.

Furthermore, the picture encoding method may include an encoding step ofencoding, for each sequence, information indicating the specified sizefor encoding the said sequence including a plurality of encodedpictures.

With this structure, a picture structure to be used for predictiveencoding can be defined flexibly for each sequence since the size isspecified for each sequence.

Also, in the management step, one of the decoded pictures judged to befor reference in the first judgment step and the decoded picture judgedto be for output in the second judgment step may be stored in an areastoring a picture that is not for reference and is judged as alreadyoutputted in the fourth judgment step when the picture buffer doesn'thave an area capable of newly storing a picture.

With the structure described above, a new picture is stored in an areastoring the outputted picture out of areas allocated for a picture foroutput in the picture buffer. Namely, the area storing the outputtedpicture is reused so that the display of the decoded picture that is notyet outputted may be more assured.

Furthermore, in the management step, one of the decoded pictures judgedto be for reference in the first judgment step and the picture judged tobe for output in the second judgment step may be stored in an areastoring a picture whose display order is the earliest of the picturesthat are not for reference and stored within the picture buffer when thepicture buffer doesn't have an area capable of newly storing a picture.

With the structure described above, there is a high possibility that adecoded picture already outputted is stored in an area storing a picturewhose display order is the earliest of the pictures for output.Therefore, the display of the decoded picture that is not yet outputtedmay be more assured although the structure does not allow a correctdetermination on the display time of respective decoded pictures.

The picture encoding method may further comprise a releasing step ofreleasing an area other than an area storing a picture which is storedbefore a size of the input picture is modified and is not partly orwholly damaged after being stored out of areas storing display pictures,when the size modification takes place in a sequence including aplurality of encoded pictures.

With the structure described above, the pictures that are not yetoutputted at the time of size modification can be outputted as many aspossible without being removed since decoded pictures for output can beleft in the decoded picture buffer when the size modification takesplace.

Thus, according to the present invention, the pictures that cannot bedisplayed correctly in the related art can be displayed by making themost use of the released memory area without having unnecessary memory,and therefore, its practical value is high.

A picture decoding method, a picture encoding apparatus, a picturedecoding apparatus and a program according to the present invention havethe same structure, operation, and effects as described above.

The picture encoding method or the picture decoding method can be any ofthe compositions described in (1) through (16) below.

(1) A picture encoding method predictively encodes a picture signal withreference to a decoded picture stored in a memory after being decoded.The picture encoding method comprises the following steps: generating apredictive image with reference to the decoded picture in the memory andencoding a difference between the predictive image and the picturesignal to obtain an encoded stream; decoding the encoded differencebetween the predictive image and the picture signal, adding thepredictive image to the decoded difference to obtain a decoded pictureand storing a predetermined decoded picture in the memory; andcontaining in the encoded stream instruction information indicatingwhether or not to release a whole area of the memory for making the areareusable.

(2) A picture encoding method predictively encodes a picture signal withreference to a decoded picture in a memory. The picture encoding methodcomprises: generating a predictive image with reference to the decodedpicture in the memory; encoding a difference between the predictiveimage and the picture signal to obtain an encoded stream; decoding theencoded difference between the predictive image and the picture signal;adding the predictive image to the decoded difference to obtain adecoded picture; and storing the decoded picture in the memory when thedecoded picture is one of i) a picture used for reference in order togenerate a predictive image and ii) a picture that is not immediatelydisplayable.

(3) A picture encoding method predictively encodes a picture signal withreference to a decoded picture. The picture encoding method comprises:generating a predictive image with reference to the decoded picture in amemory; encoding a difference between the predictive image and thepicture signal to obtain an encoded stream; decoding the encodeddifference between the predictive image and the picture signal, addingthe predictive image to the decoded difference to obtain a decodedpicture; storing a predetermined decoded picture in the memory andreleasing an unnecessary picture in the memory, wherein thepredetermined decoded picture is stored in an area storing a picturealready displayed in the released area within the memory.

(4) A picture decoding method predictively decodes an encoded streamwith reference to a decoded picture stored in a memory after beingdecoded. The picture decoding method comprises the following steps of:generating a predictive image with reference to the decoded picture inthe memory, adding the predictive image to a picture signal obtained bydecoding the encoded stream to obtain a decoded picture and storing apredetermined decoded picture in the memory; and a processing step ofperforming processing based on instruction information indicatingwhether or not to release a whole area of the memory obtained bydecoding the encoded stream for making the area reusable.

(5) A picture decoding method predictively decodes an encoded streamwith reference to a decoded picture. The picture decoding methodcomprises: generating a predictive image with reference to the decodedpicture in a memory; decoding the encoded stream; adding the predictiveimage to the decoded encoded stream to obtain a decoded picture; storinga predetermined decoded picture in the memory and releasing anunnecessary picture in the memory, wherein the predetermined decodedpicture is stored in an area in which a picture already displayed hasbeen stored in the released area within the memory.

(6) A picture encoding method predictively encodes a picture signal withreference to a decoded picture. The picture encoding method comprises:generating a predictive image with reference to the decoded picture in amemory; encoding a difference between the predictive image and thepicture signal to obtain an encoded stream; decoding the encodeddifference between the predictive image and the picture signal, addingthe predictive image to the decoded difference to obtain a decodedpicture; storing a predetermined decoded picture in the memory andreleasing an unnecessary picture in the memory, wherein thepredetermined decoded picture is stored in an area in which a picturedisplayed at the earliest time has been stored in the released areawithin the memory.

(7) A picture decoding method predictively decodes an encoded streamwith reference to a decoded picture. The picture decoding methodcomprises: generating a predictive image with reference to the decodedpicture in a memory; decoding the encoded stream; adding the predictiveimage to the decoded picture to obtain a decoded picture; storing apredetermined decoded picture in the memory and releasing an unnecessarypicture in the memory, wherein the predetermined decoded picture isstored in an area in which a picture displayed at the earliest time hasbeen stored in the released area within the memory.

(8) A picture encoding method predictively encodes a picture signal withreference to a decoded picture. The picture encoding method comprises:generating a predictive image with reference to the decoded picture in amemory; encoding a difference between the predictive image and thepicture signal to obtain an encoded stream; decoding the encodeddifference between the predictive image and the picture signal, addingthe predictive image to the decoded difference to obtain a decodedpicture; storing a predetermined decoded picture in the memory andreleasing an unnecessary picture in the memory, wherein thepredetermined decoded picture is stored in an area that is capable ofstoring a picture and is firstly released in the memory.

(9) A picture decoding method predictively decodes an encoded streamwith reference to a decoded picture. The picture decoding methodcomprises: generating a predictive image with reference to the decodedpicture in the memory; decoding the encoded stream; adding thepredictive image to the decoded encoded stream to obtain a decodedpicture; storing a predetermined decoded picture in the memory andreleasing an unnecessary picture in the memory, wherein thepredetermined decoded picture is stored in an area that is capable ofstoring a picture and is released firstly in the memory.

(10) A picture encoding apparatus predictively encodes a picture signalwith reference to a decoded picture. The picture encoding apparatuscomprises: a prediction unit operable to generate a predictive imagewith reference to the decoded picture in a memory; an encoding unitoperable to encode a difference between the predictive image predictedby the prediction unit and the picture signal to obtain an encodedstream; a decoding unit operable to decode the encoded difference; anaddition unit operable to add the predictive image to the differencedecoded by the decoding unit; a reference picture control unit operableto determine to store one of i) a picture used for reference in order togenerate a predictive image and ii) a picture that is not immediatelydisplayable; and a memory unit operable to store the adding resultobtained from the addition unit based on the determination made by thereference picture control unit.

(11) A picture encoding apparatus predictively encodes a picture signalwith reference to a decoded picture. The picture encoding apparatuscomprises: a prediction unit operable to generate a predictive imagewith reference to the decoded picture in a memory; an encoding unitoperable to encode a difference between the predictive image predictedby the prediction unit and the picture signal to obtain an encodedstream; a decoding unit operable to decode the encoded difference; anaddition unit operable to add the predictive image to the differencedecoded by the decoding unit; a reference picture control unit operableto determine whether or not to store an adding result obtained from theaddition unit and whether or not to release an unnecessary picture inthe memory; and a memory unit operable to store the adding resultobtained from the addition unit in an area in which a picture alreadyoutputted as a decoded picture has been stored in the released areawithin the memory based on the determination made by the referencepicture control unit.

(12) A picture decoding apparatus predictively decodes an encoded streamwith reference to a decoded picture. The picture decoding apparatuscomprises: a decoding unit operable to decode the encoded stream; aprediction unit operable to generate a predictive image with referenceto the decoded picture in a memory; an addition unit operable to add thepredictive image to the difference between the predictive image and thepicture signal decoded by the decoding unit; a reference picture controlunit operable to determine whether or not to store the adding resultobtained from the addition unit in the memory and whether or not torelease an unnecessary picture in the memory; and a memory unit operableto store an adding result obtained from the addition unit in an area inwhich a picture already outputted as a decoded picture has been storedin the released area in the memory based on the determination made bythe reference picture control unit.

(13) A picture encoding apparatus predictively encodes a picture signalwith reference to a decoded picture. The picture encoding apparatuscomprises: a prediction unit operable to generate a predictive imagewith reference to the decoded picture in a memory; an encoding unitoperable to encode a difference between the predictive image predictedby the prediction unit and the picture signal to obtain an encodedstream, a decoding unit operable to decode the encoded difference; anaddition unit operable to add the predictive image to the differencedecoded by the decoding unit; a reference picture control unit operableto determine whether or not to store the adding result obtained from theaddition unit in the memory and whether or not to release an unnecessarypicture in the memory; and a memory unit operable to store the addingresult obtained from the addition unit in an area in which a picturedisplayed at the earliest time has been stored in the released areawithin the memory based on the determination made by the referencepicture control unit.

(14) A picture decoding apparatus predictively decodes an encoded streamwith reference to a decoded picture. The picture decoding apparatuscomprises: a decoding unit operable to decode the encoded stream; aprediction unit operable to generate a predictive image with referenceto the decoded picture in a memory; an addition unit operable to add thepredictive image to the difference between the predictive image and thepicture signal decoded by the decoding unit; a reference picture controlunit operable to determine whether or not to store, in the memory, theadding result obtained from the addition unit and whether or not torelease an unnecessary picture in the memory; and a memory unit operableto store the adding result obtained from the addition unit in an area inwhich a picture displayed at the earliest time has been stored in thereleased area within the memory based on the determination made by thereference picture control unit.

(15) A picture encoding apparatus predictively encodes a picture signalwith reference to a decoded picture. The picture decoding apparatuscomprises: a prediction unit operable to generate a predictive imagewith reference to the decoded picture in a memory; an encoding unitoperable to encode a difference between the predictive image predictedby the prediction unit and the picture signal to obtain an encodedstream; a decoding unit operable to decode the encoded differencebetween the predictive image and the picture signal decoded by thedecoding unit; an addition unit operable to add the predictive image tothe difference decoded by the decoding unit; a reference picture controlunit operable to determine whether or not to store the adding resultobtained from the addition unit in the memory and whether or not torelease an unnecessary picture in the memory; and a memory unit operableto store the adding result obtained from the addition unit in an areathat is released at the earliest time of the whole released area withinthe memory based on the determination made by the reference picturecontrol unit.

(16) A picture decoding apparatus predictively decodes an encoded streamwith reference to a decoded picture. The picture decoding apparatuscomprises: a decoding unit operable to decode an encoded stream; aprediction unit operable to generate a predictive image with referenceto the decoded picture in a memory; an addition unit operable to add thepredictive image to the difference between the predictive image and thepicture signal decoded by the decoding unit; a reference picture controlunit operable to determine whether or not to store, in the memory, theadding result obtained from the addition unit and whether or not torelease an unnecessary picture in the memory; and a memory unit operableto store the adding result obtained from the addition unit in an areathat is released at the earliest time of the whole released area withinthe memory based on the determination made by the reference picturecontrol unit.

A storage medium for storing a program in which the pictureencoding/decoding method of the present invention is executed by acomputer may be any of (17) through (23) below.

(17) A storage medium employs a picture encoding method in which acomputer is made to predictively encode a picture signal with referenceto a decoded picture. The storage medium comprises: generating apredictive image with reference to a decoded picture in a memory;encoding a difference between the predictive image and the picturesignal to obtain an encoded stream; decoding the encoded differencebetween the encoded predictive image and the picture signal, and addingthe predictive image to the decoded difference and then outputting theadding result as a decoded picture, storing the decoded picture in thememory when the decoded picture is one of i) a picture used forreference in order to generate a predictive image and ii) a picture thatis not immediately displayable.

(18) A storage medium employs a picture encoding method in which acomputer is made to predictively encode a picture signal with referenceto a decoded picture. The storage medium comprises: generating apredictive image with reference to the decoded picture in a memory;encoding a difference between the predictive image and the picturesignal to obtain an encoded stream; decoding the encoded difference;adding the predictive image to the decoded difference to obtain adecoded picture; storing a predetermined decoded picture in the memoryand releasing an unnecessary picture in the memory, wherein thepredetermined decoded picture is stored in an area in which a picturealready displayed has been stored in the released area within thememory.

(19) A storage medium employs a picture decoding method in which acomputer is made to predictively decode an encoded stream with referenceto a decoded picture. The storage medium comprises: generating apredictive image with reference to the decoded picture in a memory;decoding the encoded stream; adding the predictive image to the decodedencoded stream to obtain a decoded picture; storing a predetermineddecoded picture in the memory and releasing an unnecessary picture inthe memory, wherein the predetermined decoded picture is stored in anarea in which a picture already displayed has been stored in thereleased area within the memory.

(20) A storage medium employs a picture encoding method in which acomputer is made to predictively encode a picture signal with referenceto a decoded picture. The storage medium comprises: generating apredictive image with reference to the decoded picture in a memory;encoding a difference between the predictive image and the picturesignal to obtain an encoded stream; decoding the encoded difference;adding the predictive image to the decoded difference to obtain adecoded picture; storing a predetermined decoded picture in the memoryand releasing an unnecessary picture in the memory, wherein thepredetermined decoded picture is stored in an area in which a picturedisplayed at the earliest time has been stored in the released areawithin the memory.

(21) A storage medium employs a picture decoding method in which acomputer is made to predictively decode an encoded stream with referenceto a decoded picture. The storage medium comprises: generating apredictive image with reference to the decoded picture; decoding theencoded stream; adding the predictive image to the decoded encodedstream to obtain a decoded picture; storing a predetermined decodedpicture in the memory and releasing an unnecessary picture in thememory, wherein the decoded picture is stored in an area in which apicture displayed at the earliest time has been stored in the releasedarea within the memory.

(22) A storage medium employs a picture encoding method in which acomputer is made to predictively encode a picture signal with referenceto a decoded picture. The storage medium comprises: generating apredictive image with reference to the decoded picture in a memory;encoding a difference between the predictive image and the picturesignal to obtain an encoded stream; decoding the encoded difference;adding the predictive image to the decoded difference to obtain adecoded picture; storing a predetermined decoded picture in the memoryand releasing an unnecessary picture in the memory, wherein thepredetermined decoded picture is stored in an area that is capable ofstoring a picture and is released firstly of the whole released area inthe memory.

(23) A storage medium employs a picture encoding method in which acomputer is made to predictively encode a picture signal with referenceto a decoded picture. The storage medium comprises: generating apredictive image with reference to the decoded picture in a memory;encoding a difference between the predictive image and the picturesignal to obtain an encoded stream; decoding the encoded difference;adding the predictive image to the decoded difference to obtain adecoded picture; storing a predetermined decoded picture in the memoryand releasing an unnecessary picture in the memory, wherein thepredetermined decoded picture is stored in an area in which a picturedisplayed at the earliest time has been stored in the released areawithin the memory.

A picture encoding method of defining memory constraint conditions for astream and a picture decoding apparatus and encoding video for thepicture encoding apparatus of the present invention may be any of (A1)through (A26) below.

(A1) A picture encoding method defines memory constraint conditions fora stream and a picture decoding apparatus. The picture encoding methodcomprises the following steps: determining a first maximum number ofreference pictures that can be used; setting a second maximum number ofreference pictures used for backward prediction; entering the secondmaximum number in a header of the stream; generating a virtual displaydelay buffer; generating a display counter; allocating a sufficientmemory space for the reference pictures; and encoding the video. Here,the virtual display delay buffer is a virtual buffer for each picturefor display to be stored in an area for display retained in a memory inthe picture encoding apparatus and stores information indicating anorder of outputting the pictures for display (picture order number orthe like). The display counter records or updates a picture order numberof the picture to be outputted virtually on the display. The sufficientmemory space indicates an area for storing both the reference picturesand the pictures for display.

(A2) The picture encoding method according to (A1), wherein the firstmaximum number is determined according to a level value indicating acapacity of the decoder.

(A3) The picture encoding method according to (A1) or (A2), wherein thesecond maximum number is smaller than the first maximum number ofreference pictures.

(A4) The picture encoding method according to (A1), (A2) or (A3),wherein the backward prediction of the entire stream is limited by thesecond maximum number.

(A5) The picture encoding method according to (A1), wherein a size ofthe virtual display delay buffer is the second maximum number minus 1.

(A6) The picture encoding method according to (A1) or (A5), wherein aphysical memory for the virtual display delay buffer includes onlydisplay order information of the pictures.

(A7) The picture encoding method according to (A1), wherein the displaycounter is used for storing display order information of the pictures tobe removed from the virtual display delay buffer.

(A8) The picture encoding method according to (A1), comprising thefollowing steps of: predicting a predictive image for a current pictureto be coded with reference to usable reference pictures; encoding thecurrent picture; determining whether or not the encoded picture can beused for prediction; decoding the encoded picture; storing the decodedpicture in the reference picture buffer; updating the virtual displaydelay buffer; and updating the display counter.

(A9) The picture encoding method according to (A1) or (A8), wherein oneor a plurality of reference pictures can be used for the prediction ofthe picture when the picture is predictively encoded.

(A10) The picture encoding method according to (A1) or (A8), wherein thereference picture can be used only when the reference picture buffermeets the memory constraint conditions.

(A11) The picture encoding method according to (A1) or (A8), wherein thereference picture is decoded when the current picture is to be used as areference picture.

(A12) The picture encoding method according to (A1), (A8) or (A10),further comprising the following steps of: determining memory constraintconditions of the number of reference pictures with respect to afollowing picture; removing an unused reference picture from thereference picture buffer; and adding the decoded picture to thereference picture buffer.

(A13) The picture encoding method according to (A1), (A8), (A10) or(A12), wherein a maximum number of the reference pictures with respectto the following picture equals to what the number of pictures in thevirtual display delay buffer is subtracted from the first maximum numberfor the stream.

(A14) A picture encoding method according to (A1), (A8), (A10) or (A12),wherein the display order information for the display of the unusedreference picture is moved to the virtual display delay buffer when thedisplay order information for the display is greater than a valueindicated at the display counter.

(A15) The picture encoding method according to (A1) or (A8), wherein thevirtual display delay buffer stores the temporal information of thepicture when a value indicated in the display order information isgreater than a value indicated at the display counter.

(A16) The picture encoding method according to (A1), (A8) or (A15),wherein the virtual display delay buffer removes the display orderinformation of the picture when the value indicated in the display orderinformation is smaller than or equal to the value indicated at thedisplay counter.

(A17) The picture encoding method according to (A1), (A8), (A10), (A12)or (A13), wherein the number of pictures in the virtual display delaybuffer is defined according to a total number indicated in the displayorder information of the pictures that are present in the virtualdisplay delay buffer.

(A18) The picture encoding method according to (A1) or (A8), wherein thedisplay counter starts updating the display order information in one ofcases: when the virtual display delay buffer is full and when the numberof encoded pictures equals to the first maximum number.

(A19) The picture encoding method according to (A1), (A8) or (A18),wherein the display counter updates the display order information basedon the display order information of each encoded picture.

(A20) The picture encoding method according to (A1), wherein a picturedecoding method employed by a picture decoding apparatus comprises thefollowing steps of: determining a maximum post decoder buffer sizeaccording to a stream header; determining a minimum number of necessaryreference pictures; allocating a sufficient physical memory space forthe reference pictures; decoding the stream; and outputting the stream.

(A21) The picture encoding method according to (A1) or (A20), whereinthe maximum post decoder buffer size is the second maximum number set bythe picture encoding apparatus.

(A22) The picture encoding method according to (A1) or (A20), whereinthe minimum number of necessary reference pictures is fixed according toa capacity level of a picture decoding apparatus that can decode thesame stream.

(A23) The picture encoding method according to (A1) or (A20), whereinthe physical memory space allocated by the picture decoding apparatuscan be used for storing both reference pictures and post decoderpictures.

(A24) The picture encoding method according to (A1) or (A20), wherein atotal number of reference pictures and post decoder pictures does notsurpass the allocated physical memory space.

(A25) The picture encoding method according to (A1) or (A20), wherein afirstly decoded picture is outputted in one of following cases: when thenumber of decoded pictures equals to the maximum size of post decoderbuffer minus 1 and when a total number of decoded pictures equals to thedetermined minimum number of reference pictures.

(A26) The picture encoding method according to (A1) or (A20), whereinthe decoded picture is removed from the post decoder buffer once thedecoded picture is sent for output.

Also, a method of decoding a stream based on memory constraintconditions for a stream and a picture decoding apparatus employed by thepicture decoding apparatus of the present invention may be any of (A27)through (A37) below.

(A27) A picture decoding method decodes a stream based on memoryconstraint conditions for a stream and a picture decoding apparatus. Thepicture decoding method comprises the following steps of: determining afirst maximum number of reference pictures used for a stream;determining a second maximum number of reference pictures used forbackward prediction; calculating a minimum size of a post decoderbuffer; allocating a sufficient physical memory space for the postreference picture buffer; allocating a sufficient physical memory spacefor the post decoder buffer; decoding the stream; storing the decodedpicture in a reference picture buffer; storing the decoded picture inthe post decoder buffer; outputting the decoded picture. Here, the postdecoder buffer is an area for display in a reference picture memory inthe picture decoding apparatus.

(A28) The picture decoding method according to (A27), wherein the firstmaximum number is fixed for the picture decoding apparatus according toa capacity level to decode the stream.

(A29) The picture decoding method according to (A27), wherein the secondmaximum number is defined as the first maximum number minus 1.

(A30) The picture decoding method according to (A27), wherein theminimum size of the post decoder buffer is defined as the second maximumnumber minus 1.

(A31) The picture decoding method according to (A27) or (A30), whereinthe minimum size of the post decoder buffer is obtained from the stream.

(A32) The picture decoding method according to (A27) or (31), whereinthe second maximum number is defined as the minimum size of the postdecoder buffer plus 1.

(A33) The picture decoding method according to (A27), wherein theminimum size of the reference picture buffer is defined as the firstmaximum number.

(A34) The picture decoding method according to (A27) or (A33), whereinthe reference picture buffer is used only for storing referencepictures.

(A35) The picture decoding method according to (A27), (A30) or (A31),wherein the post decoder buffer is used for storing non-referencepictures.

(A36) The picture decoding method according to (A27), (A32) or (A33),wherein a reference picture is removed from the reference picture bufferwhen the reference picture is judged as a non-reference picture.

(A37) The picture decoding method according to (A27), (A30), (A31) or(A35), wherein picture in the post decoder buffer is removed from thepost decoder buffer when outputted.

With the structure described above, a correct decoding of the stream canbe assured by operating both the stream and the picture decodingapparatus under the same memory constraint conditions. Otherwise, amemory space sufficient for the picture decoding apparatus to decode thestream in the process of decoding cannot be assured.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram showing a prediction structure, a decoding order anda display order of pictures.

FIG. 2 is a diagram showing a prediction structure, a decoding order anda display order of the pictures.

FIG. 3 is a block diagram showing a picture encoding apparatus forrealizing a picture encoding method of the present invention describedin a first embodiment.

FIG. 4 is a flowchart showing an operation of a reference picturecontrol unit of the present invention described in the first embodiment.

FIGS. 5A, 5B and 5C are state diagrams showing a storage status of thepictures in the memory.

FIG. 6 is a flowchart showing an operation of the picture encodingapparatus of the present invention described in a second embodiment.

FIG. 7 is a flowchart showing an operation of the picture encodingapparatus of the present invention described in a third embodiment.

FIG. 8 is a flowchart showing an operation of the picture encodingapparatus of the present invention described in a fourth embodiment.

FIG. 9 is a block diagram showing a picture decoding apparatus forrealizing a picture decoding method of the present invention describedin a fifth embodiment.

FIG. 10 is a flowchart showing an operation of the picture decodingapparatus of the present invention described in the fifth embodiment.

FIG. 11 is a flowchart showing another operation of the picture decodingapparatus of the present invention described in the fifth embodiment.

FIG. 12 is a flowchart showing yet another operation of the picturedecoding apparatus of the present invention described in the fifthembodiment.

FIG. 13 is a flowchart showing another operation of the picture decodingapparatus of the present invention described in the fifth embodiment.

FIG. 14 is a block diagram showing a usage of a virtual display delaybuffer of a picture encoding apparatus.

FIG. 15 is a block diagram showing a processing of post decoder bufferoperation for encoding according to the present invention.

FIG. 16 is a block diagram showing a processing of post decoder bufferoperation for decoding according to the present invention.

FIG. 17 is an example of using the virtual display delay buffer of thepicture encoding apparatus for limiting the maximum number of thereference pictures.

FIG. 18 is an example of using the virtual display delay buffer fordeciding the time to display a first picture.

FIGS. 19A, 19B and 19C are illustrations of a storage medium in order tostore a program for realizing the picture encoding method and thepicture decoding method of each embodiment in a computing system,described in a seventh embodiment.

FIG. 20 is a block diagram showing an overall structure of a contentsupply system described in an eighth embodiment.

FIG. 21 is an outline view showing an example of a cell phone using thepicture encoding/decoding method of the present invention described inthe eighth embodiment.

FIG. 22 is a block diagram of the cell phone.

FIG. 23 is a block diagram showing an example of a digital broadcastingsystem described in the eighth embodiment.

FIG. 24 is a diagram showing a prediction structure, a decoding orderand a display order of the pictures.

FIG. 25 is a diagram showing a prediction structure, a decoding orderand a display order of the pictures.

FIG. 26 is a block diagram of the picture decoding apparatus forrealizing the conventional picture encoding method.

FIG. 27 is a mapping diagram showing examples of codes for memorycontrol commands MMCO.

FIG. 28 is a block diagram of the picture decoding apparatus forrealizing the conventional picture decoding method.

DETAILED DESCRIPTION OF THE INVENTION

The following describes a first embodiment of the present invention.

First Embodiment

FIG. 3 is a block diagram showing a picture encoding apparatus forrealizing a picture encoding method of the present invention. The samereference numbers are used for the devices that operate in the samemanner as described in the block showing a picture encoding apparatusfor realizing a conventional encoding method shown in FIG. 26 and theirexplanation is thereby abbreviated.

Differences between the block diagram in FIG. 26 showing the pictureencoding apparatus for realizing the conventional picture encodingmethod and the block diagram in FIG. 3 showing the picture encodingapparatus for realizing the picture encoding method according to thepresent invention are that a display picture control unit DisPicCtrl isadded to FIG. 3 and that instructions sent from the display picturecontrol unit DisPicCtrl are outputted to a reference picture controlunit RefPicCtrl and a picture memory PicMem2.

In the picture encoding apparatus 100 shown in FIG. 3, a picture sizemodification unit PicSize for obtaining a picture size modified byexternal operations as well as an encoding type of each picture(I-picture, P-picture and B-picture) from the picture structuredetermination unit PicStruct and outputting information indicating thepicture size to be modified to a reference picture control unitRefPicCtrl is newly set.

The operation of the picture memory PicMem2 is almost the same as thatof the picture memory PicMem1. Therefore, only different operations willbe explained.

The display picture control unit DispPicCtrl obtains a picture time Timeand judges whether a picture, which is not stored as it is not forreference, can be displayed immediately or not (whether it is necessaryto store the picture in the picture memory until its display time). Thepicture time Time, a signal for specifying a picture to be displayed, isinputted from outside. The picture time can be obtained in the followingways: from the time information outputted from the system fortransmitting pictures via a transmission line such as a packet, from thetime information in a process of formatting a video stream and audiostream for multiplexing them; or from the time information in a processof formatting a video stream. The picture time may be either an absolutetime which informs of the time for each picture or a relative time whichinforms of the order of the pictures. Moreover, the intervals ofdisplaying pictures are normally fixed. Therefore, the order ofdisplaying pictures may be considered as the display time.

Now, the case in which the picture is immediately displayable is a casein which the picture gained after the calculation in the adder Addcorresponds with the picture to be displayed indicated by the picturetime Time. In this case, a picture to be displayed before the picturethat is not yet displayed and outputted for encoding is not found in thepicture memory PicMem2. When the picture is not immediately displayable,the display picture control unit DispPicCtrl instructs the referencepicture control unit RefPicCtrl to store the picture, even though it isnot for reference, in the picture memory PicMem2. Consequently, thepicture which is not displayed immediately is stored in the picturememory PicMem2 without fail regardless of whether it is for reference ornot and can be displayed out of the picture memory PicMem2 in thedecoding apparatus.

FIG. 4 is a flowchart showing an operation of the reference picturecontrol unit RefPicCtrl of the present invention.

The reference picture control unit RefPicCtrl judges whether or not adecoded picture (picture) is to be stored for reference for a predictiveimage (Step 10). When the decoded picture is to be used for reference,the operation proceeds to Step 12, and otherwise to Step 11.

In Step 11, the reference picture control unit RefPicCtrl judges whetheror not the decoded picture is immediately displayable. “Immediatelydisplayable” here means that the decoded picture can be displayed at thetime of being decoded (see, for example, Picture 1 in FIG. 1). Thedecoded picture which is not immediately displayable means that it needsto be displayed later (for instance, B5 shown in FIG. 1). When thepicture is immediately displayable, the operation is terminated.Otherwise, it goes on to Step 12.

In Step 12, the reference picture control unit RefPicCtrl obtains anarea capable of storing a picture in the picture memory PicMem2 andinstructs it to store the decoded picture in the area obtained in thepicture memory PicMem2 using a memory control command MMCO in Step 13.

In this way, the picture which is not displayed immediately is stored inthe picture memory PicMem2 and can be outputted for display from thepicture memory PicMem2 when the time has come for its display. This doesnot require an unnecessary picture memory assigned for a picture fordisplay and a picture that needs to be stored for display can be storedin the picture memory assigned for a picture for reference.

The picture memory PicMem2 includes an area for reference in which areference picture is stored for generating a predictive image and anarea for display in which a picture for display is stored.

Meanwhile, a picture size can be modified for each GOP (Group OfPicture) mentioned before. The modification of the picture size takesplace only when a whole area for reference in the memory storing anunnecessary reference picture is released (make the status of the memoryreusable).

However, when the modification of the picture size takes place asdescribed above, the picture for display that is not yet displayed isstored in the memory area, and it is necessary to determine explicitly acoping strategy of how to handle this picture for display but not yetdisplayed (whether to delete it or to store it until it is displayed).

Here, a storing state of pictures in the memory when the change of thepicture size takes place is explained in stages.

FIGS. 5A, 5B and 5C are state diagrams showing the storing status of thepictures in the memory in stages.

In 5A, pictures 200 a, 200 b and 200 c are the pictures for reference(the pictures to be used for reference in order to generate a predictiveimage) whereas pictures 201 a, 201 b, 201 c, 201 d, and 201 e are thepictures for display (pictures to be displayed and not displayed yet).

The pictures 201 a, 201 b, 201 c, 201 d and 201 e will be displayed inthe numeric order as shown in FIG. 5A.

FIG. 5A illustrates the status in which the whole memory areas assignedfor the reference pictures 200 a, 200 b and 200 c are released for thereusability.

FIG. 5B shows that the picture size is modified following the statusshown in FIG. 5A. A reference picture 202 a being modified to a biggersize is stored in the memory area in which the reference picture 200 ahas been stored. Furthermore, the picture for display 201 a is outputtedfor display and its memory area is released.

FIG. 5C shows a status in which the memory area storing the picture fordisplay 201 b is released after the status shown in FIG. 5B. A referencepicture 202 b being modified into a bigger size is stored in the memoryarea in which the pictures for display 201 a and 201 b have been storedand a small memory area 203 is left. Even though the memory area storingthe picture for display 201 c is released, the reference picture (whosepicture size is enlarged) cannot be newly stored.

Thus, when the picture size is modified, the pictures of differentpicture sizes are mixed in the memory (the reference pictures whosepicture sizes are enlarged and the pictures for display which are notyet displayed and whose sizes are not yet modified).

Consequently, the memory is used fragmentarily, which produces a smallmemory area that cannot be used, and the usability is therebydeteriorated. When the data in the memory is repositioned so that thesmall memory area caused by the modification of the picture sizedisappears, the memory access increases greatly and thereby it isdifficult to realize encoding and decoding operations in actual time.

When the picture size is modified, two methods are conceivable. Thefirst method is to release the area for display in which the picturesfor display that are not yet displayed are stored and the area forreference in which the reference pictures are stored (as a reusablestate) and give up the display of the pictures for display that are notyet displayed. This can prevent the fragmentary use of the memory causedby the mixture of the pictures of different sizes and thereby thedeterioration in the usability of the memory can be reduced.

The modification in the picture size described above takes place asdescribed in the following. The picture size modification unit PicSizeshown in FIG. 3 receives the encoding type (I-picture, P-picture andB-picture) of each picture determined by the picture structuredetermination unit PicStruct and the picture size for the modificationinputted from outside and outputs to the reference picture control unitRefPicCtrl an instruction to modify the picture size with the timing tostart encoding an I-picture. The I-picture is a special I-picture (IDR(Instantaneous Decoding Refresh) picture) to be inserted, for example,in the beginning of the GOP.

The second method is to switch between a method to release the wholearea of the memory and discard the pictures for display that are not yetdisplayed and a method to release only the area for reference in whichthe reference pictures are stored and display the pictures for displaythat are not yet displayed before the size modification takes place witha judgment made by a picture decoding apparatus (decoder) for decodingan encoded signal (stream) that is referred to later on so that thedisplay of the pictures for display that are not yet displayed is notobligatory. In this case, the picture decoding apparatus displays thedisplayable pictures, for instance, the undamaged pictures according tothe display order.

For operating such switching, instruction information (flag) indicatingone of the following methods: the method to release the whole area ofthe memory; and the method to release only the area for reference inwhich the reference picture is stored, or other identifiable informationis contained in the stream Str outputted from the picture encodingapparatus 100.

On the side of the picture decoding apparatus, the processing isoperated based on the instruction information placed in the stream.

The following examples are conceivable for the judging criteria in orderto switch between the two methods indicated by the instructioninformation: a content creator may decide the method according to anapplication; and only the area for reference is released but not thearea for display storing a picture for display that is not yet displayed(not releasing the whole area of the memory) when memory can afford toprovide the space.

With the above construction, the picture encoding apparatus can berealized to solve the existing problems.

Second Embodiment

The following describes a second embodiment of the present invention.

In the present embodiment, the display picture control unit DispPicCtrlshown in FIG. 3 instructs the picture memory PicMem2 not to store a newpicture in the area storing the picture that is not yet displayed, whena picture is stored in the released memory area. Normally, even when anarea for picture is released, a picture previously stored can bereproduced as long as a picture is not newly stored (overwritten) in thearea. Even if a memory area in which the picture that is not yetdisplayed is released, the picture that is not yet displayed and isreleased at the time of display, but is left without being overwritten,can be displayed by storing a new picture not in the memory area, but inthe area where the picture that is already displayed is stored. Thepicture in the released picture area of the picture memory is calledpicture for display. “Already displayed” here is practically synonymouswith “already outputted to a display device”.

FIG. 6 is a flowchart showing an operation of the picture encodingapparatus 100 of the present embodiment. The present embodimentcharacterizes in determining the storage of the picture by judgingwhether the picture stored in the released area in the memory is alreadydisplayed or not.

Firstly, in Step 20, the picture encoding apparatus 100 judges whetheror not the decoded picture is to be stored in the picture memory PicMem2based on the instructions indicated in the memory control command MMCO.

In the case of storing the decoded picture in the picture memory, thereleased picture area is obtained (Step 21) and whether a picture storedin the released picture area is already displayed or not is verified(Step 22). When it is not yet displayed, the operation goes back to Step21 and the processing continues until the released area in which thepicture already displayed has been stored is found.

When such released area is found, the decoded picture is stored in thearea (Step 23).

Thus, when the picture to be displayed is displayed, the picture that isnot yet displayed is stored in the memory without being overwrittenuntil its display time since the area storing the displayed picture isreused.

Whether a picture stored in the memory is already displayed or not canbe judged by the display picture control unit DispPicCtrl managinginformation on whether or not the picture is displayed.

Whether a picture area is a released area or not can be judged byreferring to the information on whether or not each picture area isreleased, for instance, “used (usable as a reference picture)” or“unused (no longer used as a reference picture)” stored in the picturememory PicMem2 according to the memory control command MMCO.

With the above structure, the picture encoding apparatus of the presentinvention can be realized so as to overcome the existing problems.

Thus, overwriting of a new picture to the picture that is not yetdisplayed can be prevented so that the latter picture can be outputtedfor display from the area that is already released but not overwrittenat the display time. The picture that needs to be stored for display canbe stored without requiring an unnecessary memory.

As the operation in the case which requires the modification in thepicture size is the same as in the first embodiment, the explanation isabbreviated.

Third Embodiment

The following describes a third embodiment of the present invention.

FIG. 7 is a flowchart of an operation of the picture encoding apparatus100 of the third embodiment of the present invention. The presentembodiment characterizes in determining the storage of the pictureaccording to the time when the memory is released.

Firstly, in Step 30, the picture encoding apparatus 100 judges whetheror not the decoded picture is to be stored in the picture memory PicMem2based on the instruction indicated in the memory control command MMCO.

In the case of storing the decoded picture in the picture memory, thereleased picture memory area whose display time is the earliest withinreleased area is obtained (Step 31) and the decoded picture is stored inthe obtained area (Step 32).

The memory area in which the picture is decoded and stored at theearliest time instead of the memory area released at the earliestdisplay time may be allocated as an area to store the picture. For,there is a high possibility that the pictures stored in these memoryareas are already displayed.

These earliest times are not necessarily based on time and may be theearliest time according to an order, for instance, it may be theearliest time according to a display order. For example, it is highlypossible that a picture whose display order is the earliest is alreadydisplayed and a memory area storing such picture can be reused in turnas a storing area in the memory regardless of whether it is alreadydisplayed or not. Normally, the intervals of displaying pictures isregular, and therefore, the order of displaying pictures may beconsidered as the display time.

Thus, overwriting of a new picture onto the picture that is not yetdisplayed can be prevented so that the latter picture can be outputtedfor display from the area that is already released but not overwrittenat the display time. The picture that needs to be stored for display canbe stored without requiring an unnecessary memory. The processing ofstoring the picture which needs to be stored for display can be carriedout regardless of whether the picture stored in the picture memoryPicMem2 is already displayed or not. When the earliest time indicatesthe earliest time in display time, whether or not the picture is storedat the earliest time can be judged by the display picture control unitDispPicCtrl managing the information on whether or not the picture isdisplayed.

As an operation in a case in which a modification of a picture size isrequired is the same as the one described in the first embodiment, theexplanation is abbreviated.

Fourth Embodiment

The following illustrates a fourth embodiment of the present invention.

FIG. 8 is a flowchart showing an operation of the picture encodingapparatus 100 according to the fourth embodiment of the presentinvention. The present embodiment characterizes in determining a storageof the picture according to an order of releasing the memory.

Firstly, in Step 40, the picture encoding apparatus 100 judges whetheror not the decoded picture is to be stored in the picture memory PicMem2based on the instruction indicated in the memory control command MMCO.

In the case of storing the decoded picture in the picture memory, thepicture memory area released at the earliest time within the releasedarea is obtained (Step 41) and the decoded picture is stored in theobtained area (Step 42).

Since it is a system that manages the information on display time, it isdifficult to obtain the display time depending on a method to fabricatea video decoding apparatus. It is highly possible that the picture withthe earliest display time is stored in the area firstly released ratherthan in the area released at the later time. Namely, it is with highpossibility that the picture firstly released is already displayed.Therefore, there is a high possibility that the picture that is not yetdisplayed is not overwritten when the decoded picture is stored in thepicture area firstly released.

Whether the picture area is a released area or not can be judged byreferring to the information on whether or not each picture area isreleased, for instance, “used (usable as a reference picture)” or“unused (no longer used as a reference picture)” that is stored in thepicture memory PicMem2 according to the memory control command MMCO. Or,the usage of the picture area can be fixed beforehand to a predeterminedprocedure in order to judge whether the picture area is the one that isfirstly released according to the procedure. For example, this can bejudged by the picture memory PicMem2 functioning as a memory using afirst-in first-out (FIFO) method by which the contents of record whosetime records that picture is to be stored newly are sequentiallydiscarded in turn and the latest pictures of a fixed number of frames(or number of pictures) are always stored. Thus, overwriting a newpicture to a picture that is not yet displayed can be prevented so thatthe latter picture can be outputted for display from the area that isalready released but not overwritten at the display time.

As an operation in the case in which the modification of the picturesize is required is the same as the one described in the firstembodiment, the explanation is abbreviated.

Fifth Embodiment

The following illustrates a fifth embodiment of the present invention.

FIG. 9 is a block diagram of a picture decoding apparatus for realizingthe picture decoding method of the present invention. The same referencenumbers are used for the devices that operate in the same manner as apicture encoding apparatus for realizing the picture encoding methodaccording to the present invention shown in the block diagram of FIG. 3and a picture decoding apparatus for realizing the conventional picturedecoding method shown in the block diagram of FIG. 28 and theirexplanation is thereby abbreviated.

The difference between the picture decoding apparatus 150 shown in FIG.9 and the example of conventional apparatus shown in FIG. 28 is that apicture size modification detection unit PicSizeDet and a displaypicture control unit DispPicCtrl are newly set. The picture sizemodification detection unit PicSizDet outputs a command to modify thepicture size in the timing of starting encoding of a special I-picture(IDR picture) based on the picture size for the modification obtainedfrom outside and the memory control command MMCO obtained from thevariable length decoding unit VLD.

The display picture control unit DispPicCtrl instructs the picturememory PicMem2 not to store a new picture in the area for the picturethat is not yet displayed when storing the new picture in the releasedarea. Normally, even though an area for a picture is released, data of apicture that is stored right before is left in the memory and can bereproduced as long as a picture is not newly stored (overwritten) in thearea. The picture that is released at the time of display but leftwithout being overwritten can be displayed by storing a new picture inthe area where the picture that is already displayed is stored even ifthe memory area in which the picture that is not yet displayed isreleased. “Already displayed” here is practically synonymous with“already outputted to a display device”.

Thus, when the picture to be displayed is displayed, the picture that isnot yet displayed is stored in the memory without being overwrittenuntil its display time since the area storing the displayed picture isreused.

With the above structure, the picture decoding apparatus of the presentinvention can be realized in order to overcome the existing problems.The picture decoding apparatus 150 may decode an encoded signal that isencoded by the picture encoding apparatus 100 shown in the third andfourth embodiments. In this case, the operation for the use of thepicture memory is as same as the one described for the picture encodingapparatus shown in the third and fourth embodiments. Namely, for thereusability of the picture area that is released (no longer used forreference) in the picture memory, either method of deciding the area forthe picture to be newly stored described in the first through thirdembodiments may be employed.

The following describes methods of storing the picture in the memorywhen decoding the encoded signal encoded by the picture encodingapparatus shown in the third and fourth embodiments.

Firstly, the method of storing the picture in the memory at the time ofdecoding the encoded signal encoded by the picture encoding apparatusshown in the third embodiment is explained.

The picture decoding apparatus 150 judges whether or not the decodedpicture outputted from the addition unit Add in the picture decodingapparatus 150 is to be stored in the picture memory PicMem2 based on theinstruction indicated in the memory control command MMCO.

When storing the decoded picture in the picture memory, the picturememory area firstly released is obtained and the decoded picture isstored in the obtained area.

Here, the memory area in which the picture is decoded and stored at theearliest time or the memory area in which the picture with the earliestdisplay time is stored instead of the memory area that is released atthe earliest time may be allocated as an area to store the picture. For,there is a high possibility that the pictures stored in these memoryareas are already displayed.

Since it is a system that manages the information on display time, it isdifficult to obtain the display time depending on a method to fabricatea picture decoding apparatus. It is highly possible that the picturewith the earliest display time is stored in the area in which thepicture is stored at the earliest time rather than in the one in whichthe picture is stored at the later time. It is with high possibilitythat the picture stored at the earliest time is already displayed sinceit is natural that the picture with early display time is alreadydisplayed. Therefore, there is a high possibility that the picture thatis not displayed yet is not overwritten when the decoded picture isstored in the area in which the picture stored at the earliest time hasbeen stored. When the earliest time corresponds with the earliest timewhen the picture is encoded, whether the picture is encoded at theearliest time, for instance, can be judged by the prediction structureand the decoding order of the picture.

These earliest times are not necessarily based on time and may be theearliest time according to an order. For instance, it may be theearliest time according to a display order. For example, it is highlypossible that a picture whose display order is the earliest is alreadydisplayed and a memory area storing such picture can be reused in turnas a storing area in the memory regardless of whether it is alreadydisplayed or not. Normally, the intervals of displaying pictures areregular, therefore, the order of displaying pictures may be consideredas the display time.

Thus, overwriting of a new picture onto the picture that is not yetdisplayed can be prevented so that the latter picture can be outputtedfor display from the area that is already released but not overwrittenat the display time. Also, the picture that needs to be stored fordisplay can be stored without requiring an unnecessary memory.

The processing of storing the picture which needs to be stored fordisplay can be carried out regardless of whether the picture stored inthe picture memory PicMem2 is already displayed or not. When theearliest time indicates the earliest time in display time, whether ornot the picture is stored at the earliest time can be judged by thedisplay picture control unit DispPicCtrl managing the information onwhether or not the picture is displayed.

The following describes a method of storing a picture in the memory whendecoding the encoded signal encoded by the picture encoding apparatusshown in the fourth embodiment.

The picture decoding apparatus 150 judges whether or not the decodedpicture outputted from the addition unit Add mentioned above of thepicture decoding method 150 is to be stored in the picture memoryPicMem2 based on the instruction indicated in the memory control commandMMCO.

In the case of storing the decoded picture in the picture memory, thepicture memory firstly released is obtained and the decoded picture isstored in the obtained area.

Since it is a system that manages the information on display time, it isdifficult to obtain the display time depending on a method to fabricatea video decoding apparatus. It is highly possible that the picture withthe earliest display time is stored in the area firstly released ratherthan in the area released at the later time. Namely, it is likely thatthe picture firstly released is already displayed. Therefore, there is ahigh possibility that the picture that is not yet displayed is notoverwritten by storing the decoded picture in the picture area firstlyreleased.

Whether the picture area is a firstly released picture area or not canbe judged by referring to the information on whether or not each picturearea is released, for instance, “used (usable as a reference picture)”or “unused (no longer used as a reference picture)”, that is stored inthe picture memory PicMem2 according to the memory control command MMCO,including the order of releasing the pictures. Or, the usage of thepicture area can be fixed beforehand to a predetermined procedure inorder to judge whether the picture area is the one that is firstlyreleased according to the procedure. For example, this can be judged bythe picture memory PicMem2 functioning as a memory using a first-infirst-out (FIFO) method by which the contents of record with oldrecording time are sequentially discarded when the new data is recordedso that the latest pictures of a fixed number of frames (or number ofpictures) are always stored.

Thus, overwriting of a new picture to a picture that is not yetdisplayed can be prevented and the picture can be outputted for displayfrom the area that is already released but not overwritten at thedisplay time.

The display unit Disp may be installed in the exterior of the picturedecoding apparatus 150 instead of being included in it so that thepicture decoding apparatus 150 may send only the data necessary fordisplaying a picture at the display unit Disp.

Here, an operation of the picture decoding apparatus 150 when thepicture size is modified, as already explained in the second embodiment,is illustrated. However, the operation is the same as the one describedin the third and the fourth embodiments, and therefore, the explanationis abbreviated.

A method corresponding to the first method in the case in which thepicture size is modified shown in the second embodiment is that thepicture decoding apparatus 150 releases a whole area in the memoryincluding the area for reference in which the reference picture isstored and the area for display in which the picture for display isstored and performs initialization when receiving a command to modifythe picture size.

FIG. 10 is a flowchart showing an operation of the picture encodingapparatus 150.

The picture encoding apparatus 150 judges whether or not a signaloutputted from the picture size modification detection unit PicSizeDetindicates a modification of a picture size (Step 100). When it does (Yesin Step 100), the picture encoding apparatus 150 releases a whole memoryarea of the picture memory PicMem2 (as a reusable state), initializes it(Step 102) and ends the processing.

When the picture size is not modified (No in Step 100), the picturedecoding apparatus 150 judges whether or not the memory control commandMMCO outputted from the variable length decoding unit VLD indicates therelease (initialization) of the whole memory area in the picture memoryPicMem2 (Step 101) and if it does (Yes in Step 101), releases the wholememory area (Step 102), and if not (No in Step 101), releases only thearea for reference in which the reference picture is stored (Step 103)and ends the processing.

Thus releasing the whole area of the memory prevents the fragmentary useof the memory caused by the mixture of the pictures of different sizesand thereby the deterioration in the usability of the memory can bereduced.

Next, an operation of the picture decoding apparatus 150 correspondingto the second method in the case of the modification of the picture sizeshown in the second embodiment is explained.

In the case of modifying the picture size, the methods are switched fromone to the other: a method of releasing the whole area of the memory ora method of releasing only the area for reference in which the referencepicture is stored and displaying the displayable picture out of thepictures that are not yet displayed with the judgment made by thepicture decoding apparatus 150.

As described in the second embodiment, the instruction information(flag) indicating either to release the whole memory area or to releaseonly the area for reference is contained in the memory control commandMMCO in the stream Str outputted from the picture encoding apparatus100.

The picture decoding apparatus 150 determines either of the two methodsdescribed above based on the instruction information contained in thememory control command MMCO.

FIG. 11 is a flowchart showing a determination operation.

Firstly, the picture decoding apparatus 150 judges whether or not theinstruction information (flag) indicates the initialization of the wholearea of the memory (Step 150). When it indicates the initialization ofthe whole area (Yes in Step 150), the picture decoding apparatus 150releases the whole area, for initialization, including the area fordisplay storing the picture for display that is not yet displayed (Step151), and otherwise (No in Step 150), releases only the area forreference for initialization (Step 152).

When the instruction information (flag) does not indicate theinitialization of the whole area, the picture decoding apparatus 150releases only the area for reference. As for the picture that is notdisplayed yet and stored in an area other than the area for reference,the picture decoding apparatus 150 judges the picture as such anddisplays a displayable picture.

The following describes the operation of the picture decoding apparatus150 in the case of initializing only the area for reference, withreference to a flowchart.

FIG. 12 is a flowchart showing an operation of the picture decodingapparatus 150 (decoder).

The picture decoding apparatus 150 judges whether or not the signaloutputted from the picture size modification detection unit PicSizeDetindicates a modification of a picture size (Step 200). Unless itindicates the modification of the picture size (No in Step 200), thepicture decoding apparatus 150 displays the picture to be displayed thatis not yet displayed stored in the area for display (Step 203).

Meanwhile, when the signal indicates the modification of the picturesize (Yes in Step 200) the picture decoding apparatus 150 determineswhether or not the picture for display stored in the area for display isone before the modification of the size takes place (Step 201). When itis not the case (No in Step 201), the picture decoding apparatus 150displays the picture to be displayed that is not yet displayed (Step203). When it is the case (Yes in Step 201), the picture decodingapparatus 150 judges whether or not the picture for display isdisplayable or not and displays it based on the judgment (Step 202)

The following describes an operation of the picture decoding apparatus150 for a judgment on whether the picture can be displayed or not.

FIG. 13 is a flowchart showing the operation of the picture decodingapparatus 150 with regard to a display of the picture.

In FIG. 13, the same marks are used for the same processing as describedin FIG. 12.

The picture decoding apparatus 150 judges whether or not the picture fordisplay stored in the area for display is one before the modification ofthe size takes place (Step 201). When the picture is not one before themodification takes place (No in Step 201), the picture decodingapparatus 150 displays the picture. On the other hand, when the pictureis one before the modification takes place (Yes in Step 201), thepicture decoding apparatus 150 determines whether the picture to bedisplayed is damaged or not (Step 211). When the picture is not damaged(No in Step 211) the picture is considered as displayable and therebydisplayed (Step 210). When it is damaged (Yes in Step 211), thepreceding picture, for instance, is displayed instead. Here, not beingdamaged means that the picture to be displayed is not overwritten evenpartially, so that new picture data is stored. In this way, the picturedecoding apparatus 150 judges whether or not the picture to be displayedis damaged, determines the undamaged picture as displayable and displaysit.

Thus, as explained in the flowcharts of FIGS. 11, 12 and 13, the picturedecoding apparatus 150 displays the picture that is not displayed yetaccording to the instruction information (flag) indicating whether ornot to initialize the whole area of the memory when the modification ofthe picture size takes place.

Owing to the picture decoding method of the picture decoding apparatus150 as described above, an appropriate decoding is realized by switchingeither to initialize the whole area of the memory or to initialize onlythe area for reference and display the picture for display that is notyet displayed even when the modification of the picture size isperformed. Namely, when the free area in the memory is small, forinstance, the whole area in the memory is initialized for reuse whereaswhen it is large, the case can be handled flexibly by allowing thedisplay of the picture that is not yet displayed after initializing onlythe area for reference.

Sixth Embodiment

The following terms are used in a sixth embodiment. Namely, a referencepicture buffer is an area combining an area for reference and the onefor display in the picture memory PicMem2 in a picture encodingapparatus. A virtual display delay buffer is a virtual buffer for eachpicture for display to be stored in the area for display that isretained in the picture memory PicMem2 in the picture encoding apparatusand stores temporal reference numbers of the pictures for display(picture numbers or the like). A reference memory buffer is an area forreference in the picture memory PicMem2 in a picture decoding apparatus.A post decoder buffer is an area for display in the picture memoryPicMem2 in the picture decoding apparatus. The temporal referencenumbers are the numbers that are assigned for the pictures according tothe order of display time and may be an equivalent of the picture timeTime.

Firstly, the explanation starts with the picture encoding apparatus. Thevirtual display delay buffer is used to limit the maximum number of thereference pictures used for predictive encoding performed by the pictureencoding apparatus.

FIG. 14 shows processing of determining a relationship between thevirtual display delay buffer and the maximum number of the backwardreference pictures. Each module in FIG. 14 represents a function blockor a processing step of a picture structure determination unitPicStruct, a reference picture control unit RefPicCtrl and a displaypicture control unit DispPicCtrl. As shown in the diagram, the maximumnumber of the reference pictures allowed, N_(R) for this profile andlevel, is determined in module 401. This value is defined for eachprofile and level and thereby the encoder encodes the value ofprofile/level, an equivalent of N_(R), as information on a stream and adecoder obtains N_(R) according to the value of profile/level in thestream. The picture encoding apparatus then sets the maximum number ofbackward reference pictures, N_(B), to be optimal for encoding a videosequence in module 402. The size of the virtual display delay buffer canbe determined based on this N_(B) value. When the N_(B) is less than 2,the virtual display delay buffer is not required. However, when N_(B) isgreater than or equal to 2, a virtual display delay buffer that canstore N_(B)−1 number of pictures is created in module 404. Theinformation on the picture stored in the virtual display delay buffer isretained in the memory or in any register. This virtual display delaybuffer does not require a large physical memory space in the pictureencoding apparatus. This is because only the descriptions of thereference pictures (picture numbers or the like) for identifying not awhole part of the reconstructed (decoded) picture but whichreconstructed picture are stored in the virtual display delay buffersince the judgment to see which picture is stored in the virtual displaydelay buffer for its display unless the picture decoded by the picturedecoding apparatus needs to be displayed (outputted). Besides thevirtual display delay buffer, a display counter is created in module 405and the value is retained either in the memory or in any register. Thedisplay counter is used to judge whether to remove an unnecessarypicture from the virtual display delay buffer. The picture encodingapparatus then generates a picture size for N_(R) number of referencepictures in the memory space based on the profile and level definition.Maximum virtual display delay buffer size=N _(B)−1  (1),where N_(B)<N_(R)

FIG. 15 shows the post encoding processing in the picture encodingapparatus, that is, a processing of storing a picture decoded afterbeing encoded (referred to as an encoded picture or simply a picture) inthe reference memory buffer. The module in the diagram represents afunction block or a processing step in the reference picture controlunit RefPicCtrl and the display picture control unit DispPicCtrl. Aftera single picture is encoded in module 501, whether or not the picture isto be used as a reference picture is determined in module 502. When thepicture is a reference picture, the maximum number of the possiblereference pictures is calculated based on equation (2).NMax=N _(R) −Fv  (2)

Here, NMax represents the maximum number of the possible referencepictures whereas N_(R) represents the maximum number of the referencepictures allowed in the profile and level definition. Fv represents avirtual display delay buffer fullness, that is, the number presentlyused out of the size of the virtual display delay buffer (N_(B)−1).

When the number of the reference pictures in the reference picturebuffer is less than NMax, the encoded picture is reconstructed (decoded)in module 506 and stored in the reference picture buffer in module 507.When there isn't sufficient space, the picture encoding apparatusretains a necessary area in the reference picture buffer by removingcertain unused reference pictures (no longer used as reference pictures)from the buffer, as shown in module 504. When there are no unusedreference pictures to be removed from the buffer, the encoded picture isnot used as a reference picture. When the encoded picture is not areference picture, the temporal reference number of the encoded pictureis compared with the one indicated at the display counter in module 505.When the temporal reference number of the non-reference picture is lessthan the one indicated at the display counter, the virtual display delaybuffer is updated in module 508. In module 508, the virtual displaydelay buffer removes the picture having the temporal reference numberequal to or earlier than the one indicated at the display counter andadds the temporal reference number of the current non-reference pictureinto the buffer. The number of the pictures in the virtual buffer havingthe temporal reference number less than the one indicated in the displaycounter becomes the fullness of the buffer. The display counter onlystarts updating operation with initiative when the number of the encodedpicture is equal to N_(B) or when the virtual display counter is full,no matter which may be earlier. After that, the display counter isupdated for every picture that is encoded in module 509.

FIG. 16 shows the post decoding processing in the picture decodingapparatus, that is, processing of storing a picture decoded by thepicture decoding apparatus (referred to as a decoded picture or simply apicture) in the reference picture buffer. The modules in the diagramshow the function blocks or the processing steps in the display picturecontrol unit DispPicCtrl. A picture is decoded in module 601. In module602, the picture decoding apparatus determines whether or not thepicture needs to be stored as a reference picture. If the decodedpicture is to be stored as a reference picture, it is stored in thereference picture buffer in module 606. Otherwise, the picture decodingapparatus examines whether it is time to display (output) this picturein module 603. If the time has not come yet for the picture to bedisplayed (outputted), the picture is stored in the post decoder bufferin module 604 until its display time. Both the post decoder buffer andthe reference picture buffer share physically the same memory area. Inother words, each area of the same memory can be used as a referencepicture buffer for some occasions and as a post decoder buffer for otheroccasions.

In module 605, the picture decoding apparatus then determines a pictureto be displayed (outputted) either from the reference picture buffer orthe post decoder buffer based on the display timing. Once a picture fromthe post decoder buffer is displayed (outputted), it is removed from thebuffer. On the other hand, a reference picture is only to be removedfrom the reference picture buffer or shifted to the post decoder bufferwhen the stream indicates that the reference picture is no longer usedfor reference.

FIG. 17 is an example of a method using a virtual display delay bufferfor determining the maximum number of reference pictures at each pictureinterval. In this example, the maximum number of reference picturesallowed, N_(R), shall be 4. The picture encoding apparatus sets themaximum number of backward reference pictures, N_(B), to be 3. Thus,virtual display delay buffer in which two pictures are storable isgenerated (the area is retained). The display counter is used forstoring and updating the temporal reference number of the picture to bevirtually displayed (outputted). In the reference memory buffer shown inthe diagram, the vacant area shows that the area is vacant, the areadescribed with its picture type and the number shows that the picturefor reference is stored, and the hatched area shows that the picture fordisplay is stored. Also, in the virtual display delay buffer, the vacantarea shows that the area is vacant, the area described with its picturetype and the number shows that the temporal reference number of thepicture for display in the reference memory buffer is stored and thehatched area shows that the reference memory buffer does not have astoring area for the picture for display. The virtual display delaybuffer whose number corresponds to the number of the hatched area in thereference memory buffer stores the temporal reference numbers of thepictures for display.

As shown in the diagram, the pictures B2 and B3 are used as referencepictures and thereby they are stored in the reference memory buffertogether with I0 and P4, and the display counter starts updating afterthe picture B3 is encoded. At time T6, B5 is not used as a referencepicture; however, this is because the temporal reference number isgreater than the one indicated at the display counter and needs to beupdated by the virtual display delay buffer. Therefore, the fullness ofthe virtual display delay buffer Fv at that time is 1 and the maximumnumber of possible reference pictures NMax is 3.

Similarly at time T7, the picture B6 needs to be updated by the virtualdisplay delay buffer whereas B5 cannot be removed yet due to itstemporal reference number being greater than the one indicated at thedisplay counter. Thus, NMax at that time indicates 2. The referencepicture I0 has to be removed from the reference memory buffer due to theinsufficient memory. B7 at that time then can be predicted only with theuse of P4 and P8.

FIG. 18 shows an example in which the display counter is firstlyincremented using the virtual display delay buffer. N_(R) shall be 5 inthe example. The picture encoding apparatus sets the maximum number ofmultiple backward reference pictures N_(B) to be 3. Thus a virtualdisplay delay buffer in which two pictures are storable is generated.

As shown in the diagram, B1, B2, B3 and B4 are not used for referencepictures whereas B7 and B8 are. B1 and B2 are therefore stored in thevirtual display delay buffer at time T3 in order to wait for display.Since the virtual display delay buffer is full at time T3, the displaycounter starts updating at time T3. The reason why the display counterneeds to wait at least until the virtual display delay buffer becomesfull before it can start updating is that it needs to handle thereordering of the B-pictures that is likely to occur in the picturedecoding apparatus.

As shown in FIG. 17, the picture decoding apparatus sometimes uses apart of the reference picture buffer as a post decoder buffer. Thepicture decoding apparatus therefore operates the reference picturebuffer based on a fixed physical memory size of the reference picturebuffer so that some parts of the reference picture buffer are used forreference pictures and the rest is used as the post decoder buffer. Forexample, at time T3, the whole reference picture buffer is used to storereference pictures whereas at time T7, only two areas of the referencepicture buffer are used to store the reference pictures of P4 and P8.The rest is used to store post decoder pictures B5 and B6.

The maximum number of the reference pictures to be used at the picturedecoding apparatus side is defined in the profile and level definitionfor the picture encoding apparatus. The picture encoding apparatustherefore can use up to the maximum number of the reference picturesdefined by the profile and level. Thus, it is required that the pictureencoding apparatus controls the reference pictures and performs encodingso that the same operation can be realized (the number of referencepictures does not surpass the predetermined value) when the picturedecoding apparatus operates based on the constraint conditions.

Similarly, the picture decoding apparatus has the same number ofreference pictures as the picture encoding apparatus does. In addition,additional memory space is needed for the post decoder buffer. Themaximum number of post decoder pictures is defined by equation (3).N _(P) =N _(R)−2  (3)

Here, N_(P) presents the maximum number of post decoder pictures whereasN_(R) presents the maximum number of the reference pictures defined bythe profile and level definition.

The maximum number of post decoder pictures shall be thereforeconsidered in the design of the picture decoding apparatus so that thepicture decoding apparatus complies with the profile and leveldefinition. The maximum number of the post decoder pictures can beeither calculated with the use of the equation (3) or specified in theprofile and level definition. Once the maximum number of post decoderpictures is specified in the profile and level definition, the maximumnumber of backward prediction pictures can be calculated based onequation (4).N _(B) =N _(P)+1  (4)

Here, N_(B) presents the maximum number of backward prediction picturesin the reference picture buffer. In the present embodiment, N_(B)presents the minimum memory requirements required by the picturedecoding apparatus so as to decode a stream complying with the profileand level definition.

It is possible to limit the number of pictures used for backwardprediction in order to reduce the number of the additional post decoderpictures added to the reference picture buffer by the side of thepicture decoding apparatus. For example, when the maximum number of thepictures used for backward prediction is limited to 2, only oneadditional post decoder picture is required to be added to the referencepicture buffer, which can prevent the allocation of unnecessary physicalmemory space for the post decoder pictures on the side of the picturedecoding apparatus.

Seventh Embodiment

Furthermore, it is possible to easily perform the processing shown inthe above embodiments in an independent computing system by recording aprogram for realizing the picture encoding method and the picturedecoding method shown in the above-mentioned embodiments onto a storagemedium such as a flexible disk.

FIGS. 19A, 19B and 19C are illustrations for realizing the pictureencoding/decoding method of the above embodiments using a programrecorded onto a storage medium like a flexible disk.

FIG. 19B shows a full appearance of a flexible disk, its structure atcross section and the flexible disk itself, whereas FIG. 19A shows anexample of a physical format of the flexible disk as a main body of astoring medium. A flexible disk FD1 is contained in a case F, aplurality of tracks Tr are formed concentrically from the periphery tothe inside on the surface of the disk, and each track is divided into 16sectors Se in the angular direction. Therefore, as for the flexible diskstoring the above-mentioned program, data as the aforementioned programis stored in an area assigned for it on the flexible disk FD1.

FIG. 19C shows a structure for recording and reading out the program onthe flexible disk FD1. When the program is recorded on the flexible diskFD1, the computing system Cs writes in data as the program via aflexible disk drive FDD. When the picture encoding method and thepicture decoding method for realizing the picture encoding method andthe picture decoding method as the program on the flexible disk areconstructed in the computing system, the program is read out from theflexible disk by the flexible disk drive FDD and then transferred to thecomputing system Cs.

The above explanation is made on an assumption that a storing medium isa flexible disk, but the same processing can also be performed using anoptical disk. In addition, the storing medium is not limited to aflexible disk and an optical disk, but any other medium such as an ICcard and a ROM cassette capable of recording a program can be used.

Eighth Embodiment

The following is an explanation of the applications of the pictureencoding method as well as the picture decoding method as shown in theabove-mentioned embodiments, and a system using them.

FIG. 20 is a block diagram showing an overall configuration of a contentsupply system ex100 for realizing content distribution service. The areafor providing communication service is divided into cells of desiredsize, and cell sites ex107˜ex110 which are fixed wireless stations areplaced in respective cells.

This content supply system ex100 is connected to devices such asInternet ex101, an Internet service provider ex102, a telephone networkex104, as well as a computer ex111, a PDA (Personal Digital Assistant)ex112, a camera ex113, a cell phone ex114 and a cell phone with a cameraex115 via the cell sites ex107˜ex110.

However, the content supply system ex100 is not limited to theconfiguration as shown in FIG. 20 and may be connected to a combinationof any of them. Also, each device may be connected directly to thetelephone network ex104, not through the cell sites ex107˜ex110.

The camera ex113 is a device capable of shooting video such as a digitalvideo camera. The cell phone ex114 may be a cell phone of any of thefollowing system: a PDC (Personal Digital Communications) system, a CDMA(Code Division Multiple Access) system, a W-CDMA (Wideband-Code DivisionMultiple Access) system or a GSM (Global System for MobileCommunications) system, a PHS (Personal Handyphone System) or the like.

A streaming server ex103 is connected to the camera ex113 via thetelephone network ex104 and also the cell site ex109, which realizes alive distribution or the like using the camera ex113 based on the codeddata transmitted from the user. Either the camera ex113 or the serverwhich transmits the data may code the data. Also, the picture data shotby a camera ex116 may be transmitted to the streaming server ex103 viathe computer ex111. In this case, either the camera ex116 or thecomputer ex111 may code the picture data. An LSI ex117 included in thecomputer ex111 or the camera ex116 actually performs encodingprocessing. Software for encoding and decoding pictures may beintegrated into any type of storage medium (such as a CD-ROM, a flexibledisk and a hard disk) that is a recording medium which is readable bythe computer ex111 or the like. Furthermore, a cell phone with a cameraex115 may transmit the picture data. This picture data is the dataencoded by the LSI included in the cell phone ex115.

The content supply system ex100 encodes contents (such as a music livevideo) shot by a user using the camera ex113, the camera ex116 or thelike in the same way as shown in the above-mentioned embodiments andtransmits them to the streaming server ex103, while the streaming serverex103 makes stream distribution of the content data to the clients attheir requests. The clients include the computer ex111, the PDA ex112,the camera ex113, the cell phone ex114 and so on capable of decoding theabove-mentioned encoded data. In the content supply system ex100, theclients can thus receive and reproduce the encoded data, and can furtherreceive, decode and reproduce the data in real time so as to realizepersonal broadcasting.

When each device in this system performs encoding or decoding, thepicture encoding method or the picture decoding method shown in theabove-mentioned embodiments can be used.

A cell phone will be explained as an example of the device.

FIG. 21 is a diagram showing the cell phone ex115 using the pictureencoding method and the picture decoding method explained in theabove-mentioned embodiments. The cell phone ex115 has an antenna ex201for communicating with the cell site ex110 via radio waves, a cameraunit ex203 such as a CCD camera capable of shooting moving and stillpictures, a display unit ex202 such as a liquid crystal display fordisplaying the data such as decoded pictures and the like shot by thecamera unit ex203 or received by the antenna ex201, a body unitincluding a set of operation keys ex204, an audio output unit ex208 suchas a speaker for outputting audio, a audio input unit ex205 such as amicrophone for inputting audio, a storage medium ex207 for storing codedor decoded data such as data of moving or still pictures shot by thecamera, data of received e-mails and that of moving or still pictures,and a slot unit ex206 for attaching the storage medium ex207 to the cellphone ex115. The storage medium ex207 stores in itself a flash memoryelement, a kind of EEPROM (Electrically Erasable and Programmable ReadOnly Memory) that is a nonvolatile memory electrically erasable from andrewritable to a plastic case such as an SD card, or the like.

Next, the cell phone ex115 will be explained with reference to FIG. 22.In the cell phone ex115, a main control unit ex311, designed in order tocontrol overall each unit of the main body which contains the displayunit ex202 as well as the operation keys ex204, is connected mutually toa power supply circuit unit ex310, an operation input control unitex304, a picture encoding unit ex312, a camera interface unit ex303, anLCD (Liquid Crystal Display) control unit ex302, a picture decoding unitex309, a multiplexing/demultiplexing unit ex308, a read/write unitex307, a modem circuit unit ex306, and an audio processing unit ex305via a synchronous bus ex313.

When a call-end key or a power key is turned ON by a user's operation,the power supply circuit unit ex310 supplies respective units with powerfrom a battery pack so as to activate the camera attached digital cellphone ex115 as a ready state.

In the cell phone ex115, the audio processing unit ex305 converts theaudio signals received by the audio input unit ex205 in conversationmode into digital audio data under the control of the main control unitex311 including a CPU, ROM and RAM, the modem circuit unit ex306performs spread spectrum processing of the digital audio data, and thecommunication circuit unit ex301 performs digital-to-analog conversionand frequency transform of the data, so as to transmit it via theantenna ex201. Also, in the cell phone ex115, the communication circuitunit ex301 amplifies the data received by the antenna ex201 inconversation mode and performs frequency transform and analog-to-digitalconversion to the data, the modem circuit unit ex306 performs inversespread spectrum processing of the data, and the audio processing unitex305 converts it into analog audio data, so as to output it via theaudio output unit ex208.

Furthermore, when transmitting an e-mail in data communication mode, thetext data of the e-mail inputted by operating the operation keys ex204of the main body is sent out to the main control unit ex311 via theoperation input control unit ex304. In the main control unit ex311,after the modem circuit unit ex306 performs spread spectrum processingof the text data and the communication circuit unit ex301 performsdigital-to-analog conversion and frequency transform for the text data,the data is transmitted to the cell site ex110 via the antenna ex201.

When picture data is transmitted in data communication mode, the picturedata shot by the camera unit ex203 is supplied to the picture encodingunit ex312 via the camera interface unit ex303. When it is nottransmitted, it is also possible to display the picture data shot by thecamera unit ex203 directly on the display unit ex202 via the camerainterface unit ex303 and the LCD control unit ex302.

The picture encoding unit ex312, which includes the picture encodingapparatus as explained in the present invention, compresses and codesthe picture data supplied from the camera unit ex203 by the encodingmethod used for the picture encoding apparatus as shown in theabove-mentioned first embodiment so as to transform it into encodedpicture data, and sends it out to the multiplexing/demultiplexing unitex308. At this time, the cell phone ex115 sends out the audio receivedby the audio input unit ex205 during the shooting with the camera unitex203 to the multiplexing/demultiplexing unit ex308 as digital audiodata via the audio processing unit ex305.

The multiplexing/demultiplexing unit ex308 multiplexes the encodedpicture data supplied from the picture encoding unit ex312 and the audiodata supplied from the audio processing unit ex305 using a predeterminedmethod, then the modem circuit unit ex306 performs spread spectrumprocessing of the multiplexed data obtained as a result of themultiplexing, and lastly the communication circuit unit ex301 performsdigital-to-analog conversion and frequency transform of the data for thetransmission via the antenna ex201.

As for receiving data of a moving picture file which is linked to a Webpage or the like in data communication mode, the modem circuit unitex306 performs inverse spread spectrum processing of the data receivedfrom the cell site ex110 via the antenna ex201, and sends out themultiplexed data obtained as a result of the inverse spread spectrumprocessing.

In order to decode the multiplexed data received via the antenna ex201,the multiplexing/demultiplexing unit ex308 separates the multiplexeddata into an encoded stream of picture data and that of audio data, andsupplies the encoded picture data to the picture decoding unit ex309 andthe audio data to the audio processing unit ex305 respectively via thesynchronous bus ex313.

Next, the picture decoding unit ex309, including the picture decodingapparatus as explained in the above-mentioned invention, decodes theencoded stream of picture data using the decoding method correspondingto the encoding method as shown in the above-mentioned embodiments togenerate reproduced moving picture data, and supplies this data to thedisplay unit ex202 via the LCD control unit ex302, and thus the picturedata included in the moving picture file linked to a Web page, forinstance, is displayed. At the same time, the audio processing unitex305 converts the audio data into analog audio data, and supplies thisdata to the audio output unit ex208, and thus the audio data included inthe moving picture file linked to a Web page, for instance, isreproduced.

The present invention is not limited to the above-mentioned system assuch ground-based or satellite digital broadcasting has been in the newslately and at least either the picture encoding apparatus or the picturedecoding apparatus described in the above-mentioned embodiments can beincorporated into a digital broadcasting system as shown in FIG. 23.More specifically, an encoded stream of video information is transmittedfrom a broadcast station ex409 to or communicated with a broadcastsatellite ex410 via radio waves. Upon receipt of it, the broadcastsatellite ex410 transmits radio waves for broadcasting. Then, a home-useantenna ex406 with a satellite broadcast reception function receives theradio waves, and a television (receiver) ex401 or a set top box (STB)ex407 decodes the encoded stream for reproduction. The picture decodingapparatus as shown in the above-mentioned embodiment can be implementedin a reproducing apparatus ex403 for reading out and decoding theencoded stream recorded on a storage medium ex402 that is a recordingmedium such as CD and DVD. In this case, the reproduced video signalsare displayed on a monitor ex404. It is also conceivable to implementthe picture decoding apparatus in the set top box ex407 connected to acable ex405 for a cable television or the antenna ex406 for satelliteand/or ground-based broadcasting so as to reproduce them on a monitorex408 of the television ex401. The picture decoding apparatus may beincorporated into the television, not in the set top box. Also, a carex412 having an antenna ex411 can receive signals from the satelliteex410 or the cell site ex107 for reproducing moving pictures on adisplay device such as a car navigation system ex413 set in the carex412.

Furthermore, the picture encoding apparatus as shown in theabove-mentioned embodiments can encode picture signals and record themon a recording medium. As a concrete example, a recorder ex420 such as aDVD recorder for recording picture signals on a DVD disk ex421, a diskrecorder for recording them on a hard disk, or the like can be cited.They can be recorded on an SD card ex422. If the recorder ex420 includesthe picture decoding apparatus as shown in the above-mentionedembodiments, the picture signals recorded on the DVD disk ex421 or theSD card ex422 can be reproduced for display on the monitor ex408.

As for the structure of the car navigation system ex413, the structurewithout the camera unit ex203, the camera interface unit ex303 and thepicture encoding unit ex312, out of the components shown in FIG. 22, forexample, is conceivable. The same applies for the computer ex111, thetelevision (receiver) ex401 and others.

In addition, three types of implementations can be conceived for aterminal such as the above-mentioned cell phone ex114; asending/receiving terminal implemented with both an encoder and adecoder, a sending terminal implemented with an encoder only, and areceiving terminal implemented with a decoder only.

As described above, it is possible to use the picture encoding method orthe picture decoding method described in the above-mentioned embodimentsfor any of the above-mentioned devices and systems, and by thus doing,the effects described in the above-mentioned embodiments can beobtained. Also, the present invention is not limited to theabove-mentioned embodiments and a wide range of variations ormodifications within the scope of the following claims are possible.

In the present embodiment, the term “display” means to output afterreordering the order of the pictures into a display order. Namely, aprocess for outputting a decoded picture to a display device isdescribed as “display”. An operation of outputting the decoded picturein order to record it onto other recording device, for instance, appliesto this special usage of “display”. When the display device has apicture memory, it is possible to transmit the decoded picture to thedisplay device earlier than the actual display time.

INDUSTRIAL APPLICABILITY

The present invention is used for a picture encoding apparatus forencoding a picture by a predictive encoding with reference to a decodedpicture stored in a decoded picture buffer, decoding the encoded pictureand storing the decoded picture in the decoded picture buffer and apicture encoding apparatus for decoding the picture encoded by apredictive encoding with reference to a decoded picture stored in adecoded picture buffer and storing the decoded picture in the decodedpicture buffer.

The invention claimed is:
 1. A moving picture coding and decoding systemwhich includes a moving picture coding apparatus which predictivelycodes a moving picture comprising a plurality of pictures and a movingpicture decoding apparatus which decodes a coded stream obtained bypredictively coding the moving picture, wherein the moving picturecoding apparatus comprises: a first coding unit operable to encode aninput picture to obtain a coded input picture; a first decoding unitoperable to decode the coded input picture to obtain a decoded picture;a picture buffer operable to store the decoded picture when the decodedpicture is a reference picture; and a second coding unit operable tocode, on a sequence basis, a buffer size information indicating that apicture buffer has a predetermined size, the sequence being a unit ofsuccessive plural pictures included in the moving picture, wherein themoving picture decoding apparatus comprises: a buffer size informationobtaining unit operable to obtain the buffer size information on asequence basis by decoding the coded stream; a second decoding unitoperable to obtain a decoded picture by decoding the coded stream; afirst judging unit operable to judge whether the decoded picture is areference picture or a non-reference picture based on the coded stream;a picture buffer operable to store the decoded picture; a second judgingunit operable to judge, if the decoded picture is judged to be anon-reference picture, whether or not the decoded non-reference pictureis necessary to be stored in the picture buffer until a display time ofthe decoded non-reference picture; a storing unit: operable to store thedecoded picture in the picture buffer as a reference picture if thedecoded picture is judged to be a reference picture, and operable tostore the decoded non-reference picture in the picture buffer if thesecond judging unit judges that the decoded non-reference picture isnecessary to be stored in the picture buffer until a display time of thedecoded non-reference picture; a third judging unit operable to judgewhether the decoded reference picture stored in the picture buffer orthe decoded non-reference picture stored in the picture buffer isnecessary to be outputted for display based on the display time of thedecoded reference picture and the display time of the decodednon-reference picture; a fourth judging unit operable to judge whetheror not the decoded reference picture stored in the picture buffer can beused as a reference picture; and a decoded picture releasing unit:operable to release the decoded reference picture stored in the picturebuffer after outputting the decoded reference picture based on theresult of the fourth judging unit, and operable to release the decodednon-reference picture stored in the picture buffer after outputting thedecoded non-reference picture for display, wherein, in the storing unit,the storing of the decoded reference picture and the storing of thedecoded non-reference picture comprises re-using a same area of thepicture buffer with the predetermined size.